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(54) Method and apparatus for annotating widgets 



(57) Widgets are annotated with freeform digital ink. 
Rather than annotating the data displayed by widgets, 
annotations are made "on" the widgets themselves and 
stored in relation to the application and/or window they 



are associated with. These annotations allow users to 
interact with widgets without having to keep a separate 
hardcopy set of notes or instructions explaining how to 
use the widgets. 
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Description 

[0001] This inv ntion is related to annotating widgets 
of a graphical user interface. In particular, this invention 
is directed to a system and method of annotating widg- s 
ets to facilitate a user's learning to use a graphical user 
interface. 

[0002] Complex graphical user interfaces that contain 
windows, such as collections of menus, dialog boxes 
and toolbars found in everyday software applications, 10 
can be hard to learn. Users must often invoke long, pre- 
cise sequences of commands or dialog box interactions 
to accomplish a desired task. Although on-line help 
manuals are commonly available, on-line help manuals 
often fail to clarify the problem at hand, and usually re- * 5 
quire several interactions before the desired information 
is obtained. While users can usually determine the cor- 
rect sequence of actions through trial and error, infre- 
quently-used command sequences are often forgotten 
and thus must be subsequently relearned every time 20 
they are required. Furthermore, beginners often keep 
long sequences of notes which correlate to and supple- 
ment actions to be performed in a given software appli- 
cation. However, because these notes are separate 
from, and not particularly well-linked to, the software ap- 25 
plication, these hardcopy notes are frequently extremely 
burdensome to use. 

[0003] The APPLE GUIDE system is a mechanism in 
the computer's help system for showing a user how to 
accomplish specific tasks with widgets. Specifically, to 30 
explain some feature of the interface, highlights and ink 
marks are used to mark widgets. However, these high- 
lights only exist during the help animation. The f reeform 
digital ink marks cannot be created or manipulated by 
the user, nor are they available outside the help system. 35 
[0004] Widgets are seen throughout today's software 
applications. Widgets, being placeholders for all con- 
trols in an interface, can range from windows in a graph- 
ical user interface, to simple dropdown menus, to but- 
tons in a toolbar, to highly complex multi-level widgets 40 
that control complex functions or enable a user to select 
multiple options for a given application. Tracking a us- 
er's path through the maze of interfaces to achieve a 
desired result has become cumbersome. 
[0005] This invention provides systems and methods 4S 
for annotating widgets. 

[0006] This invention separately provides systems 
and methods that allow a user to select an annotation 
mode, and with f reeform digital ink, to record comments 
or notes directly on widgets. 50 
[0007] This invention separately provides systems 
and methods that al tow audio data to be associated with 
a widget and to be played by a user to, for example, 
direct the user how a particular function should be per- 
formed. 55 
[0008] This invention separately provides systems 
and methods that allow a user to attach video data to a 
widget and to play back the video data tof urther facilitate 
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performing a particular function. 
[0009] This invention separately provides systems 
and methods that allow a user to annotate widgets with 
ASCII text. 

[0010] This invention separately provides systems 
and methods for attaching annotations to a widget that 
can then be shared with other users. 
[001 1] In traditional f reeform digital ink interfaces, an- 
notations occur "inside" the widgets, rather than "on" the 
widgets. The methods and systems of this invention dif- 
fer because, in the systems and methods of this inven- 
tion, an annotation, e.g., f reeform digital ink, is not treat- 
ed as user data, e.g., a signature field, or as an anno- 
tation on user data, but rather as a method of associat- 
ing comment, instructions or other annotations with the 
user interface. While freeform ink has been used to cre- 
ate content and to annotate documents, it has not been 
used to annotate user interface widgets. In Interactive 
Sketching for the Early Stages of User Interface Design, 
James A. Landay, Ph.D. dissertation, Report # CMU- 
CS-96-201, Computer Science Department, Camagie 
Mellon University, Pittsburgh, PA, December 26, free- 
form digital ink is used to add widgets to user interface 
prototypes, rather than to annotate existing widgets. 
Specifically, the prototype widgets are the added con- 
tent. In contrast, the systems and methods of this inven- 
tion allow for annotating of existing widgets themselves. 
Furthermore, U.S. Patent Application (Attorney Docket 
No. 100930), herein incorporated by reference in its en- 
tirety, provides methods and systems for maintaining 
freeform ink annotations on changing views. 
[001 2] According to methods and apparatus of this in- 
vention, a user can directly annotate a widget with free- 
form graphical ink. Specifically, the user's annotations 
are recorded directly on widgets, without affecting their 
functionality. 

[0013] These and other features and advantages of 
this invention are described in or are apparent from the 
following detailed description of the preferred embodi- 
ments. 

[0014] The preferred embodiments of this invention 
will be described in detail, with reference to the following 
figures, wherein: 

Fig. 1 shows an exemplary window comprising a 
number of widgets; 

Fig. 2 shows an exemplary set of multi-level win- 
dows; 

Fig. 3 shows a widget annotated according to this 
invention; 

Fig. 4 is an exemplary widget demonstrating anno- 
tations made according to this invention; 
Fig. 5 is an exemplary set of annotated widgets cor- 
responding to the unannotated widgets shown in 
Fig. 2 annotated according to this invention; 
Fig. 6 shows an exemplary dropdown menu anno- 
tated according to this invention; 
Fig. 7 shows an exemplary dialog box; 
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Fig. 8 shows an exemplary annotated dialog box 
corresponding to the dialog box shown in Fig. 7; 
Fig. 9 shows a second ex mplary dialog box; 
Fig. 1 0 shows a second exemplary annotated dialog 
box corresponding to the dialog box shown in Fig. 9; s 
Fig. 11 shows one exemplary embodiment of afunc- 
tional block diagram of a system for annotating 
widgets according to this invention; 
Fig. 12 is an exemplary structural block diagram of 
a window; 

Fig. 1 3 is an exemplary structural block diagram of 
a widget annotated according to this invention; 
Fig. 1 4 shows an audio annotation added to a widg- 
et according to this invention; 
Fig. 1 5 shows a video annotation added to a widget 
according to this invention; 
Fig. 16 is a third exemplary annotated widget anno- 
tated according to this invention; 
Fig. 17 is a flowchart outlining one exemplary meth- 
od for annotating a widget according to this inven- 
tion; and 

Fig. 18 is a flowchart outlining another exemplary 
method for displaying an annotated widget accord- 
ing to this invention. 

[0015] To annotate widgets, the systems and meth- 
ods of this invention associate and store a user's anno- 
tations corresponding to one or more particular widgets. 
Fig. 1 is an exemplary window 100 of a graphical user 
interface. Common elements, i.e. widgets, within a win- 
dow are dropdown menus 120, checkboxes 130, and 
tabs 140. Widgets within the window, such as the widg- 
ets shown in Fig. 1 , are used to specify certain criteria 
of a desired or selected function. Although the window 
100 shown in Fig. 1 is a simple example, windows can 
become highly complicated and contain hundreds of 
widgets corresponding to control options. Fig. 2 illus- 
trates an exemplary multi-leveled window containing a 
first level window 210, a second level window 220 and 
a third level window 230. Clearly, with the quantity of 
information provided to a user in this context, it would 
be easy for the user to forget to or be unable to recognize 
the desired function. 

[0016] The systems and methods of this invention 
provide for annotating these widgets, as shown in Fig. 
3. Specifically, the annotated window 300 of Fig. 3 is an 
annotated version of the window 100 illustrated in Fig. 
1 . By way of example, the user has indicated in an an- 
notation 310 added to in the window 300 that the vertical 
ruler is "good for column inches," and in an annotation 
320 that certain scrollbars are not needed for the 
SCROLLPOINT mouse. Other annotations 330-360 
have been added to the widget 300 to provide additional 
information about the widget 300 to the user. 
[0017] Fig. 4 is an exemplary window 400 which con- 
tains an annotation 410 that indicates a course of action 
to be taken before a particular function is performed. 
[0018] Fig. 5 is an exemplary cascaded set of multi- 



level windows 500, comprising windows 510-530 which 
have been annotated to direct a user through sequential 
events. Specifically, when a certain featur of the first 
level window 510 is selected, the subsequent window 
520 is displayed. Again the user is directed via annota- 
tions added to the window 520 to a particular location 
within the window 520. Upon selecting a function within 
the annotated window 520, the user can then be direct- 
ed to the annotated or highlighted function in the window 
530. 

[001 9] Fig. 6 illustrates an annotated dropdown menu 
600. Specifically, the annotation 607 illustrates the cir- 
cled menu item should be selected for insertion of a pic- 
ture. Annotation 605 highlights that "caption" adds a 
caption to an inserted picture. Through the above ex- 
amples, it is clear that the annotations added according 
to this invention are not limited to any particular type of 
window or graphical user interface, but rather are appli- 
cable to all widgets. 

[0020] As previously mentioned, the methods and 
systems of this invention allow for annotating widgets 
themselves, rather than for annotating the data the 
widgets contain. This is illustrated with reference to Fig. 
7-10. Fig. 7 shows a print dialog box for a particular print- 
er. The displayed data 655 corresponds to the particular 
selected printer. Fig. 8 shows annotations 660 and 662 
that have been made to the print dialog box of Fig. 7. 
[0021 ] Fig. 9 shows the same print dialog box for an- 
other printer. The displayed data 665 corresponds to this 
second selected printer. Fig. 10 shows the same anno- 
tations 670 and 672 that have been made to the print 
dialog boxes of Figs. 8 and 9. 

[0022] Fig. 1 1 is a functional block diagram illustrating 
one embodiment of a widget annotation system 700 ac- 
cording to this invention. As shown in Fig. 11 , the widget 
annotation system 700 comprises a user input device 
710, such as a mouse, pen or a keyboard, a mode se- 
lector 720, a widget annotation database 730, an anno- 
tation manager 740, an annotation renderer 745, a dis- 
play 750 that displays a widget 760 to be, or that has 
been, annotated, and a window manager 755. 
[0023] A user, through the user input device 710, in- 
teracts with the system 700 to annotate the widget 760. 
The annotation manager 740 monitors annotations 
made to the widget 760. Once an annotation is made, 
the annotation manager 740 saves the annotation infor- 
mation with the associated unique widget identifier in the 
widget annotation database 730. Upon a subsequent 
display of the same widget 760. the annotation manager 
740 accesses the annotations, if any, from the widget 
annotation database 730, and the annotation renderer 
745 displays the annotations. 

[0024] The widget annotation database 730 stores 
uniqu widget identifiers with corresponding annota- 
tions. Th uniqu widget identifier as used in this appli- 
cation comprises an application identifier, a widget iden- 
tifier and an optional user identifier. The application 
identifier identifies a window manager or an application. 
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The widget identifier comprises a static resource iden- 
tifier and/or an identifier maintained dynamically by an 
application or window manager The optional user iden- 
tifier uniquely identifies a user of a system. 
[0025] The mode selector 720 controls the type of an- 
notation mode selected by the user. The display 750 can 
be a monitor or other comparable display device that is 
capable of displaying widgets 760. 
[0026] Fig. 12 is a structural representation of an ex- 
emplary window 800. The window includes a number of 
widgets 810 that represent all the visual elements form- 
ing the window 600. 

[0027] As shown in Fig. 13 t annotations 910 to the 
window 800 appear as annotations or markings on one 
or more of the individual widgets 810. These annota- 
tions or markings 910 do not alter the widget itself, but 
rather are "layered" over the assemblage of widgets 810 
that form the window 800. 

[0028] Figs. 1 4 and 1 5 illustrate additional exemplary 
types of annotations that can be added to a widget ac- 
cording to this invention. For example, the window 800 
shown in Fig. 14 contains an icon 1000 representing an 
audio segment added or attached to a widget 1010 with- 
in the window 800. As with f reeform ink annotations, the 
icon 1 000 is displayed in conjunction with the associated 
annotated widget. However, in addition to the visual con- 
tent associated with the icon itself, the icon represents 
a link, such as a hyperlink or pointer, to an audio file. 
This audio file may be a preexisting audio file, such as 
a "wav" file. Alternatively, the audio file could be record- 
ed by a user specifically for the current annotation. For 
example, when a user decides to annotate a widget, the 
user could be prompted for the type of annotation de- 
sired. If the user selects "audio annotation," for example, 
the system could record the user's audio clip or prompt 
the user for an already existing audio clip file. During 
playback, the user simply selects the icon 1000, for ex- 
ample with a mouse click. In response, an audio clip is 
played to the user relating to the content of the widget 
1010 to which the icon 1000 has been associated. Fig. 
1 5 illustrates a video icon 1 1 00 associated with a widget 
1 110. As with audio annotations, video annotations can 
be recorded or replayed when the user indicates a video 
or multimedia annotation is desired, or selects the video 
icon 1100, respectively. 

[0029] Fig. 16 is another exemplary annotated win- 
dow 1200, illustrating that annotations are not limited to 
one particular widget, but can be strewn throughout the 
entire window. 

[0030] In operation, a user selects the mode in which 
annotations are to be performed. For example, one 
mode can be activated by simply inking over the nonac- 
tive portions of the window. Alternatively, if annotations 
are desired within an active element, a user can tempo- 
rarily enter a manual mod which would disable the ac- 
tive components of the window and allow annotations 
to be placed anywhere within the window. 
[0031] In either of these modes, the annotations are 



recorded identically in the widget annotation database. 
For example, in automatic mode, if the annotation, e.g., 
inking process, event occurs over an active widget, i. . , 
a radio button, no annotation, e.g., ink mark, is made. If 

s a selection device controlled by the user, e.g., a mouse 
pointer, passes over an active widget, e.g., a radio but- 
ton, that active widget performs normally. Alternatively, 
if no widget responds to the annotation event, an anno- 
tation mode can be triggered. 

10 [0032] A second exemplary embodiment of the meth- 
ods according to this invention includes an annotation 
button in the application environment. When the button 
is selected, the widgets are disabled, the cursor option- 
ally changes to indicate the annotation mode, and user- 

15 selected events produce annotations. 

[0033] Digital ink can be used to embody the user's 
annotations. For example, the f reeform digital ink can 
be used to highlight text or to actually change the char- 
acteristics of the text, e.g., underlining. Furthermore, an- 

20 notations is not limited to actual written annotations by 
the user. The annotations may encompass audio or vid- 
eo data, animated graphics, and/or may be either trans- 
parent or opaque, depending on the user's desire. Fur- 
thermore, it should be understood that the annotations 

25 to the widget need not be static. Active elements or hy- 
perlinks can also be included to further broaden the use- 
fulness of the annotations. 

[0034] All annotations to the widget are stored in the 
widget annotation database 730. This widget annotation 

30 database 730 need not be integral to one particular us- 
er's application and one particular user's annotations. 
By storing the unique widget identifiers, the widget an- 
notation database 730 can be shared with other users 
so they can use a first user's annotations. The receiving 

35 user can preview and/or ignore the previous annota- 
tions, can replace the existing annotations with the new 
annotations, and/or add new annotations, or merge the 
two sets of annotations. 

[0035] Fig. 17 is a flowchart outlining one exemplary 
40 method for annotating a widget according to this inven- 
tion. Control begins in step S100, and then continues to 
step S200, where a widget is displayed. With the widget 
displayed, a user has the option of annotating the widg- 
et. Thus, in step S300, a user determines whether au- 
45 tomatic or manual annotation is desired. If automatic op- 
eration is desired, control continues to step S400. Oth- 
erwise, if manual annotation is desired, control jumps to 
step $700. 

[0036] In step S400, the particular annotation type de- 
so sired is selected. This can range, for example, from font 
attributes, to text attributes, to line thickness and/or 
color, to insertion of audio or video elements, plain text, 
highlighting or other type of freeform digital ink. Then, 
in step S500, the actual annotations are created and 
55 added by the user relative to the widget. Next, in step 
S600, the added annotations are associated with the 
widget and saved. Control then continues to step S700. 
[0037] In step S700, the user is queried whether an- 
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2. The method of claim 1 , wherein the annotations re- 
main associated with the widget even if information 
within the widget chang s. 

5 3. The method of claim 1 or claim 2, further compris- 
ing: 

determining a location of a cursor relative to the 
at least one widget; and 
10 enabling an annotation mode based on the de- 

termined location. 

4. The method of claim 3, wherein the annotation 
mode is activated if the cursor is located in a non- 

is active portion of the at least one widget. 

5. The method of claim 3 or claim 4, wherein the an- 
notation mode is deactivated if the cursor is located 
in an active portion of the at least one widget. 

20 

6. The method of any of the preceding claims, wherein 
associating the at least one annotation with the 
widget comprises associating a unique widget iden- 
tifier, which comprises an application identifier, with 

25 the at least one annotation. 

7. The method of any of claims 1 to 5, wherein asso- 
ciating the at least one annotation with the widget 
comprises associating a unique widget identifier, 

30 which comprises a widget identifier, with the at least 
one annotation. 

8. The method of any of claims 1 to 5, wherein asso- 
ciating the at least one annotation with the widget 

35 comprises associating a unique widget identifier, 
which comprises a user identifier, with the at least 
one annotation. 

9. A method for displaying a widget, comprising: 

40 

identifying at least one annotation associated 
with the widget that is to be displayed with the 
widget; and 

displaying the widget and the at least one as- 
45 sociated annotation. 



other annotation is desired. If another annotation is de- 
sired, control jumps back to step S300. If no further an- 
notations are desired, control continues to step S1300, 
where the control sequence ends. 
[0038] In step S800, the manual annotation mode is 
activated. This manual annotation mode allows user 
complete control over the type and placement of the an- 
notations. Next, in step S900, as with step S400, the 
user selects the annotation type. Then, in step S1000, 
the user creates and adds the annotation relative to the 
widget. Control then continues to step S1100. 
[0039] In step S1 1 00, the annotations are associated 
with the corresponding widgets and are saved. Then, in 
step S1200, the user determines if additional annota- 
tions are desired. If so, control jumps back to step S900. 
Otherwise, control continues to step S1 300 where the 
control routine ends. 

[0040] Fig. 18 is a flowchart outlining one exemplary 
embodiment of a method for displaying an annotated 
widget according to this invention. Control begins in step 
S3000, and then continues to step S31 00, where a widg- 
et is displayed. Then, in step S3200, a determination is 
made as to whether the displayed widget is annotated. 
If the displayed widget is annotated, control jumps to 
step S3210 Otherwise, control continues to step S3300. 
In step S3300, the control sequence ends. 
[0041] In step S3210, the unique widget identifier is 
determined. Next, in step S3220, the corresponding an- 
notation is loaded. Then, in step S3230, the annotation 
is displayed. Control then passes to step S3300, where 
the control sequence ends. 

[0042] As shown in Fig. 1 1 , the widget annotation sys- 
tem is preferably implemented on one or more pro- 
grammed general purpose computers. However, the 
widget annotation system can also be implemented on 
a special purpose computer, a programmed microproc- 
essor or microcontroller and peripheral integrated circuit 
elements, an ASIC or other integrated circuit, a digital 
single processor, a hardwired electronic or logic circuit 
such as a discrete circuit, a programmable logic device 
such as a PLD, PLA, PGA, FPGA or PAL, or the like. In 
general, any device, capable of implementing a finite 
state machine that is in turn capable of implementing 
the flowcharts shown in Figs. 17-19, can be used to im- 
plement the widget annotation system according to this 
invention. 



Claims 

1 . A method for annotating a widget comprising: 

enabling an annotation mode that allows at 
least one annotation that is displayable with a 
widget to be made relative to that widget; and 
associating the at least one annotation with that 
widget. 



10. The method of claim 9, further comprising updating 
data within the widget. 

50 11. The method of claim 9 or claim 10, wherein identi- 
fying the at least one associated annotation com- 
prises: 

reading at least one unique widget identifier; 
55 and 

leading at least one corresponding annotation. 

12. A widget annotation system, comprising: 
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an input device that receives at least one an- 
notation for at least one widget; and 
a widget annotation database that stores the at 
least one annotation associated with the at 
least one widget. s 

13. The system of claim 12, further comprising a mode 
selector that sets an annotation mode. 

14. The system of claim 13, wherein the annotation '0 
mode is determined by a location of a cursor relative 

to the at least one displayed widget. 

15. The system of any of claims 12 to 14, wherein the 
widget annotation database stores at least one *5 
unique widget identifier with at least one corre- 
sponding annotation. 

16. The system of any of claims 12 to 15, further com- 
prising a widget display device on which the at least 20 
one widget is displayed. 

17. A system that renders annotations associated with 
a widget, comprising: 

25 

a widget annotation database that stores at 
least one annotation associated with at least 
one widget; and 

a widget display device that displays the widget 
to be displayed and the at least one associated 30 
annotation. 

18. The system of claim 17, wherein an application up- 
dates information in the widget to be displayed. 

35 
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